Summary

FairShareAggregator summed each bucket's resource amounts and slice durations
separately, then multiplied — computing (Σ amount) × (Σ duration) instead of
Σ(amount × duration). This inflates a bucket by the number of kernel slices
folded into it (≈ concurrent kernel count), and the error compounds up the
hierarchy. Reported case: one user with 4 GPU kernels for a day reads 1382400
fGPU-seconds instead of the correct 345600. kernel_usage_records, which
stores per-slice products, was never affected and is the rebuild source.

• Aggregator now accumulates per-slice products, so the delta carries
  Σ(amount × duration) directly. BucketDelta (its only job was holding the
  two separate sums) collapses to a plain ResourceSlot.
• usage_bucket_entries.amount → resource_usage, dropping its
  NUMERIC(24,6) limit — a domain-level daily mem bucket runs to ~1e18
  byte-seconds; unbounded NUMERIC has no ceiling. Rename + typmod drop are
  metadata-only (no table rewrite). resource_usage is what the JSONB mirror and
  the three bucket tables already call this, so one term runs end to end.
• duration_seconds dropped — nothing read it on its own; it only existed to
  reconstitute the product.
• Read-path scale fix (second, independent defect): the fair-share read path
  summed the stored figure without multiplying by duration, so scheduling ran on
  a ~300× too-small scale. Storing the product directly corrects it.

Migration c4a91d7e05b2

Existing buckets are wrong in both the JSONB mirror and the normalized entries and
cannot be fixed in place, so both are rebuilt from kernel_usage_records:

• Rebuilds usage_bucket_entries + the three *_usage_buckets JSONB mirrors.
  *_fair_shares is left alone — the next observer tick recomputes it from the
  corrected entries.
• Excludes the oldest retained day (retention purges kernel records mid-day by
  period_end, so that day may be truncated). Buckets older than the
  kernel-record retention window keep their inflated values rather than being
  zeroed — unrecoverable, and zeroing would destroy the only history left. The
  ~28-day fair-share lookback sits inside the rebuildable window.
• Rebuild is idempotent: it deletes the in-window entries and re-inserts from the
  immutable kernel_usage_records, and overwrites (not increments) the JSONB, so
  re-running yields the same values.
• downgrade() reverts only the column definition (the rebuilt values are the
  correct ones; the inflated originals are unrecoverable) and logs a warning that
  the entries no longer match the restored schema.

The rebuild spans up to the retention window; back up before applying.

Test plan

• pants test on the aggregator, resource_usage_history and retention suites —
  bucket assertions corrected to resource-seconds; regression parametrized over
  1/2/4/10 concurrent kernels; multi-tenant tick (two projects, one user active in
  both) pinning per-level factors and bucket keys
• pants fmt/lint/check clean on the changeset
• Migration verified on a live DB: down→up round-trips are idempotent, every
  in-window level rebuilds to its expected value, the oldest retained day stays
  untouched, buckets with no kernel records collapse to {}, and the downgrade
  warning prints
• Verify on a live manager that new observation ticks accumulate correctly

Follow-ups (not in this PR)

• The JSONB bucket update is a read-modify-write, not a server-side accumulate, so
  concurrent observers could lose an update (only leader election guards it today).
• period_end is not in the on_conflict update set, so the period-extension
  strategy described in the bucket docstrings never actually happens.
• ResourceSlot has no unit notion, so an allocation and an already-integrated
  value share a type — what made this defect expressible; distinguishing them
  properly spans ~15 files and belongs in its own change.

Resolves BA-6927

🤖 Generated with Claude Code